Papers with manual analysis
What do Large Language Models Learn about Scripts? (2022.starsem-1)
Copied to clipboard
| Challenge: | Script Knowledge is important for language understanding but expensive to produce manually and difficult to induce from text due to reporting bias. |
| Approach: | They propose a pipeline-based script induction framework which can generate good quality ESDs for unseen scenarios. |
| Outcome: | The proposed framework produces good quality ESDs for unseen scenarios, but manual evaluation shows there is room for improvement. |
Semantic Structural Decomposition for Neural Machine Translation (2020.starsem-1)
Copied to clipboard
| Challenge: | Existing methods for translation of long sentences are limited by the translation of single sentences to single sentences. |
| Approach: | They propose to use semantic splitting of the source sentence as preprocessing for machine translation. |
| Outcome: | The proposed approach tackles two main limitations of state-of-the-art machine translation. |
How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2023.tacl-1)
Copied to clipboard
| Challenge: | Current language models generate high-quality text, but are they copying it or have they learned generalizable linguistic abstractions? |
| Approach: | They propose a suite of analyses for assessing the novelty of generated text . they focus on sequential structure (n-grams) and syntactic structure (syntactical structure). |
| Outcome: | The proposed model-generated text is as novel as the baseline human-generated model- generated text, but it is copied substantially, the authors show . |
TabGenie: A Toolkit for Table-to-Text Generation (2023.acl-demo)
Copied to clipboard
| Challenge: | TabGenie enables researchers to explore, preprocess, and analyze data-to-text generation datasets. |
| Approach: | They present TabGenie, a toolkit which enables researchers to explore, preprocess, and analyze a variety of data-to-text generation datasets. |
| Outcome: | The toolkit provides an interactive mode for debugging table-to-text generation, side-by-side comparison of generated system outputs, and easy exports for manual analysis. |
Echoes from Alexandria: A Large Resource for Multilingual Book Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent research in text summarization has focused on news stories, where texts are typically short and have strong layout features. |
| Approach: | They propose a resource for multilingual book summarization that uses a new extractive-then-abstractive baseline to compare the results. |
| Outcome: | The proposed resource is the largest and first to be multilingual, featuring 5 languages and 25 language pairs. |
End-to-End Neural Bridging Resolution (2022.coling-1)
Copied to clipboard
| Challenge: | state-of-the-art resolvers for bridging resolution are weaker than entity coreference resolution. |
| Approach: | They evaluate bridging resolvers in an end-to-end setting and strengthen them with better encoders . they also try to gain a better understanding of them through perturbation experiments . |
| Outcome: | bridging resolvers are evaluated in an end-to-end setting and strengthened with better encoders . bribridging resolution is the task of identifying briating anaphors and linking them to their antecedents - a paper by the journal bribing resolution argues . |
QTSumm: Query-Focused Summarization over Tabular Data (2023.emnlp-main)
Copied to clipboard
Yilun Zhao, Zhenting Qi, Linyong Nan, Boyu Mi, Yixin Liu, Weijin Zou, Simeng Han, Ruizhe Chen, Xiangru Tang, Yumo Xu, Dragomir Radev, Arman Cohan
| Challenge: | Existing text generation systems that can provide accurate table summaries can facilitate more efficient access to relevant data insights. |
| Approach: | They propose a query-focused task where text generation models have to perform human-like reasoning and analysis over the given table to generate a tailored table summary. |
| Outcome: | The proposed method improves existing baselines on table-to-text generation and large language models by concatenating generated facts to the model input. |
SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources with Retrieval and Semantic Parsing (2024.findings-acl)
Copied to clipboard
| Challenge: | SPAGHETTI: Semantic Parsing Augmented Generation for Hybrid English information from Text Tables and Infoboxes is a hybrid question-answering pipeline . |
| Approach: | They propose a hybrid question-answering pipeline that leverages knowledge from multiple knowledge sources. |
| Outcome: | The proposed approach achieves state-of-the-art on the Compmix dataset with 56.5% exact match rate. |
What Motivates You? Benchmarking Automatic Detection of Basic Needs from Short Posts (2021.acl-short)
Copied to clipboard
| Challenge: | According to the self-determination theory, the levels of satisfaction of three basic needs (competence, autonomy and relatedness) have implications on people’s everyday life and career. |
| Approach: | They propose to model a task that automatically detects three basic needs on short posts in English and then apply them to a binary task. |
| Outcome: | The proposed model achieves similar performance as a trained human annotator in the real-world. |
Bridging Resolution: Making Sense of the State of the Art (2021.naacl-main)
Copied to clipboard
| Challenge: | bridging resolution is a task that involves identifying and resolving bridling/associative anaphors, which are anamorphic references to non-identical associated antecedents. |
| Approach: | They propose a hybrid rule-based and MTL approach that would enable a better understanding of their comparative strengths and weaknesses. |
| Outcome: | The proposed model would be able to better understand their strengths and weaknesses and perform a manual analysis of the errors made by the model. |
Social Commonsense for Explanation and Cultural Bias Discovery (2023.eacl-main)
Copied to clipboard
| Challenge: | Social commonsense contains many human biases due to social and cultural influence. |
| Approach: | They aim to identify cultural biases in data that strongly influence model decisions . they use social commonsense knowledge to augment large-scale language models . |
| Outcome: | The proposed method shows that social commonsense knowledge can explain model behavior on two social tasks. |
Evidence-based Fact-Checking of Health-related Claims (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evidence-based factchecking datasets contain synthetic claims and lack real-world verification. |
| Approach: | They propose a dataset for evidence-based fact-checking of health-related claims that evaluates their truthfulness against scientific articles. |
| Outcome: | The proposed dataset evaluates real-world claims against scientific articles. |
Can NLI Models Verify QA Systems’ Predictions? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent question answering systems perform well on benchmark datasets, but are not always well-calibrated to spot spurious answers under distribution shifts. |
| Approach: | They propose to use natural language inference to verify whether answers are correct . they leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules. |
| Outcome: | The proposed approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. |
Transforming Complex Sentences into a Semantic Hierarchy (P19-1)
Copied to clipboard
| Challenge: | Existing approaches for recursively splitting and rephrasing complex English sentences into a semantic hierarchy of simplified sentences are lacking. |
| Approach: | They propose a method for recursively splitting and rephrasing complex English sentences into a semantic hierarchy of simplified sentences. |
| Outcome: | The proposed approach outperforms state-of-the-art approaches in MT and information extraction tasks. |
Automatic Normalisation of Early Modern French (2022.lrec-1)
Copied to clipboard
| Challenge: | Spelling normalisation is a useful step in the study and analysis of historical language texts, whether it is manual analysis by experts or automated analysis using downstream natural language processing (NLP) tools. |
| Approach: | They propose a new benchmark for the normalisation of Early Modern French into contemporary French using ABA, alignment-based approach and MT-approaches. |
| Outcome: | The proposed method homogenises the variable spelling in historical documents and reduces the gap between the historical state of the language and the contemporary state. |
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization (2021.naacl-main)
Copied to clipboard
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, Dragomir Radev
| Challenge: | Existing work on meeting summarization tasks is limited to short summaries that cover all the content of a meeting. |
| Approach: | They propose a query-based multi-domain meeting summarization task that generates a single short summary of meetings based on a transcript. |
| Outcome: | The proposed task is based on 1,808 query-summary pairs over 232 meetings in multiple domains. |
The Connection between the Text and Images of News Articles: New Insights for Multimedia Analysis (2020.lrec-1)
Copied to clipboard
| Challenge: | a case study of text and images reveals the inadequacy of simplistic assumptions about their connection and interplay. |
| Approach: | They propose to use a case study to analyze 1000 flood-related news articles . they find that articles cluster into seven categories related to different topical aspects of flooding . |
| Outcome: | The results show that flood-related news articles do not consistently report on a single, currently unfolding flooding event. |
Improving Abstractive Dialogue Summarization with Speaker-Aware Supervised Contrastive Learning (2022.coling-1)
Copied to clipboard
| Challenge: | Existing summarization systems based on pre-trained models cannot recognize the unique format of the speaker-utterance pair well in the dialogue. |
| Approach: | They propose three speaker-aware supervised contrastive learning tasks to solve the speaker identification problem in dialogue summarization task. |
| Outcome: | The proposed methods improve on two mainstream dialogue summarization datasets. |
Sequence Models for Document Structure Identification in an Undeciphered Script (2022.emnlp-main)
Copied to clipboard
| Challenge: | a systematic analysis of “header” signs in proto-Elamite provides new evidence for their existence . experts have hypothesized that headers are a sign which qualifies transactions . |
| Approach: | They provide unsupervised neural and statistical sequence modeling techniques to identify “header” signs in proto-Elamite, an undeciphered script from 3100-2900 BCE. |
| Outcome: | The authors provide new evidence for the existence of "header" signs in proto-Elamite . they examine which features predict their presence and identify correlations between features and other document properties . |
Dr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using the TREC Misinformation dataset, we empirically evaluate ChatGPT to show not just its effectiveness but reveal that knowledge passed in the prompt can bias the model to the detriment of answer correctness. |
| Approach: | They empirically evaluate ChatGPT to find out whether a prompt can bias the model to the detriment of answer correctness. |
| Outcome: | The proposed model can be biased to the detriment of answer correctness by using retrieved-then-generate pipelines and how a user phrases their question as well as the question type. |
Reassessing Semantic Knowledge Encoded in Large Language Models through the Word-in-Context Task (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have propelled significant progress, extending their application across various domains including dialogue systems, text generation, translation systems, and beyond. |
| Approach: | They propose to use the Word-in-Context (WiC) task to reassess the semantic knowledge encoded in large language models (LLMs) they prompt LLMs to generate natural language descriptions that contrast the meanings of the target word in two contextual sentences given in the WiC dataset. |
| Outcome: | The proposed model significantly improves the classification accuracy of the two models. |
How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on tau-bench (2025.findings-emnlp)
Copied to clipboard
Venkatesh Mishra, Amir Saeidi, Satyam Raj, Mutsumi Nakamura, Gaowen Liu, Ali Payani, Jayanth Srinivasa, Chitta Baral
| Challenge: | Recent advances in reasoning and planning capabilities of large language models have enabled their potential as autonomous agents capable of tool use in dynamic environments. |
| Approach: | They propose an input-reformulation multi-agent framework that reformulates user queries . |
| Outcome: | The proposed framework outperforms ReAct, Function Calling, and Self-Reflection in overall pass5 scores. |